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SYSTEM AND METHOD FOR CREATING MULTI-PRIORITY STREAMS 
BACKGROUND OF THE INVENTION 

1. Technical Field 

The present invention relates generally to systems for processing compressed 
5 video data, and more particularly to a system and method for dividing MPEG video data 
into multiple streams of different priority. 

2. Related Art 

As the demand for the on-line delivery of video applications continues to grow 
and transmission bandwidth remains at a premium, the ability to provide highly efficient 

10 transmission schemes has become indispensable. One such scheme involves dividing 

MPEG coded video data into multiple streams, each having a different priority. This type 
of scheme, referred to herein as "multi-priority streams," allows different portions of the 
video data to consume different amounts of bandwidth, depending on the importance of 
the particular portion. 

15 One specific scenario where multi-priority streams are useful involves 

applications that utilize transmission channels susceptible to errors. In these cases, it is 
necessary to protect the data with additional error protection bits, which may for example 
provide a certain amount of redundancy. Unfortunately, the inclusion of error protection 
bits creates additional overhead in terms of bandwidth consumption. By dividing MPEG 

20 data into a plurality of streams of differing priorities, individual streams can be assigned 
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different levels of error protection. Accordingly, high priority streams can receive a 
greater amount of error protection than lower priority streams. Thus, overhead can be 
balanced with the corresponding priority of the streams. 

A second scenario where multi-priority streams are useful involves networks that 
5 offer multiple quality of service (QoS) options. In such a case, different levels of quality 
of service for transmitted data are made available by the underlying transport. Thus, in 
order to efficiently utilize such a system, it is necessary to divide the video data into 
streams of differing priority, and then send each stream over the connection offering the 
appropriate QoS. 

1 0 Unfortunately, existing methods of creating multi-priority streams for MPEG 

video are limited. For example, one approach is to divide the bitstream into different 
priorities based on the type of frame. In particular, since for picture quality I frames are 
generally more important than P frames, and P frames are generally more important than 
B frames, I frames receive a highest priority, P frames receive a medium priority, and B 

15 frames receive a lowest priority. However, because the actual size of a typical I or P 
frame can be hundreds of thousands of bits, this type of prioritization provides only 
marginal results since only three priority levels can be implemented. Moreover, there can 
be a large amount of variation in terms of importance between two I frames, or two P 
frames, and such variations are not taken into account in the above scheme. Thus, a need 

20 exists to create a finer prioritization among and between frames. 

A second approach for creating multi-priority streams for MPEG video is to 
create scalable video that contains a base layer and one or more enhancement layers. The 
base layer is assigned a higher priority than the enhancement layers. The drawback of 
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this approach is the increased complexity required for encoding and decoding, and the 
need to transcode the already coded non-scalable bitstream into a scalable bitstream. 

Accordingly, a system is required: (1) that can provide a finer prioritization than 
the I, P and B frame prioritization; and (2) that can directly process a coded bitstream 
5 without incurring the complexity of a system that utilizes scalable video. 

SUMMARY OF THE INVENTION 

This invention overcomes the above-mentioned problems, as well as others, by 
providing a system and method for prioritizing streams of compressed video data suitable 
for transmission over a multi-priority transmission channel. In a first aspect, the 

1 0 invention provides a method for assigning priority to streams of compressed video data, 
comprising the steps of: determining a relative importance of each macroblock in a video 
frame based on how often each macroblock acts as a reference macroblock; and 
prioritizing each of the macroblocks in the video frame based on the relative importance. 
In a second aspect, the invention comprises a method for assigning priority to 

1 5 streams of compressed video data, comprising the steps of: determining an importance 
value for each macroblock in a plurality of video frames based on how often each 
macroblock acts as a reference macroblock; grouping macroblocks into sets of 
macroblocks, and combining the importance values of the macroblocks within each set; 
and prioritizing each set of macroblocks based on the combined importance values. 

20 In a third aspect, the invention provides a system for prioritizing streams of 

compressed video data, comprising: a system for determining an importance value for 
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each macroblock in a video frame based on how often each macroblock acts as a 
reference macroblock; and a system for prioritizing each of the macroblocks in the video 
frame based on the importance value determined for each macroblock. 

In a fourth aspect, the invention provides a system for prioritizing streams of 

5 compressed video data, comprising: a system for determining an importance value for 
each macroblock in a plurality of video frames based on how often each macroblock acts 
as a reference macroblock; a system for grouping macroblocks into sets of macroblocks 
and combining the importance values for macroblocks in each set; and a system for 
prioritizing sets of macroblocks based on the combined importance values. 

10 In a fifth aspect, the invention provides a decoder system for decoding multi- 

priority compressed video data, comprising: a system that correlates an error protection 
scheme to each of a plurality of data streams; and a system that interprets each data 
stream based on the error protection scheme; wherein the error protection scheme is 
determined by a prioritization system that prioritizes each data stream based on how often 

15 macroblocks act as reference macroblocks. 

In a sixth aspect, the invention provides a program product stored on a recordable 
media, that when executed, prioritizes streams of compressed video data, the program 
product comprising: means for determining an importance value for macroblock data in 
video frames based on how often each of a plurality of macroblocks act as reference 

20 macroblocks; and means for prioritizing macroblock data based on the determined 
importance values. 
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In a further aspect, each of the prioritization systems and methods described 
above may further include a system for assigning different error protection schemes to 
streams of different priority. 


BRIEF DESCRIPTION OF THE DRAWINGS 


5 The preferred exemplary embodiment of the present invention will hereinafter be 

described in conjunction with the appended drawings, where like designations denote like 
elements, and: 

Figure 1 depicts a block diagram of a multi-priority coding system in accordance 
with a preferred embodiment of the present invention. 
10 Figure 2 depicts a P frame analysis in accordance with the present invention. 

Figure 3 depicts an I frame analysis in accordance with the present invention. 

Figure 4 depicts an indirect analysis in accordance with the present invention. 

Figure 5 depicts a partial reference block analysis in accordance with the present 
invention. 


1 5 DETAILED DESCRIPTION OF THE INVENTION 

Referring now to Figure 1, a multi-priority coding system 10 is depicted. System 
10 includes an encoder 12 for converting a bitstream 18 of MPEG data into a set of multi- 
priority streams 16, and a decoder 14 for interpreting the set of streams 16. Encoder 12 
could be incorporated directly into an MPEG encoder, or alternatively, operate on already 
20 encoded MPEG data. Similarly, decoder 14 could be incorporated into an MPEG 
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decoder or reside separately from an MPEG decoder. Each of the multi-priority streams 
16 generated by encoder 12 are encoded with a different error protection scheme. Thus, 
for example, where the data is being transmitted over a channel susceptible to errors (e.g., 
the Internet), more important data can be afforded a greater amount of bandwidth to 
5 allow for a greater amount of error protection. On a receiving end, decoder 14 decodes 
the set of multi-priority streams 16 and outputs decoded video data 20. While this 
preferred embodiment is described as processing MPEG data, it should be understood 
that the invention could apply to any data compression scheme using predictive coding. 
Encoder 12 comprises an importance analysis system 22 that examines 

10 macroblock data and determines the relative importance of each macroblock or set of 
macroblocks. Stream prioritization system 24 then assigns a relative priority to each 
macroblock or group of macroblocks based on the above analysis. Error protection 
system 25 then encodes the macroblock data with the appropriate error protection scheme 
for transmission as one of the plurality of multi-priority streams 16. 

15 Importance analysis system 22 includes a P frame analysis system 26; an I frame 

analysis system 28; a partial macroblock analysis system 27; an indirect analysis system 
29; and a residual analysis system 19. P frame analysis system 26 and I frame analysis 
system 28 examine macroblocks within P and I frames, respectively, to determine the 
relative importance of the macroblock data. Specifically, when a P or I frame is 

20 analyzed, system 26 or 28 systematically examines each macroblock, and a relative 
importance value is calculated as each macroblock (i.e., the "current macroblock") is 
examined. Importance is based on how often the current macroblock acts as a reference 
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macroblock or partial reference block. (Note that for the purposes of this invention, the 
term "reference macroblock" may comprise a complete or partial reference block.) 

Because P and I frames are used for forward and backward prediction, P frame 
analysis system 26 and I frame analysis system 28 analyze the motion vectors of previous 
5 and subsequent B frames, and a subsequent P frame (if applicable), to determine how 
often a current macroblock within either a P or I frame acts as a reference macroblock. 
An importance value is determined based on the number of target macroblocks that 
reference the current macroblock in either the P or I frame (i.e., the number of 
predictions). Examples of this process are described below with reference to Figures 2 

10 and 3. (Note that certain P frames are followed by I frames, and therefore will not have a 
subsequent P frame to analyze.) 

It should be appreciated that importance analysis system 22 can analyze 
individual macroblocks for their relative priority or sets of macroblocks (e.g., an entire 
frame or even a set of frames such as a group of pictures). In the case where sets of 

1 5 macroblocks are being analyzed for their importance, importance analysis system 22 
would first group sets of macroblocks together based on a predetermined scheme. The 
importance value of the set is then determined by combining (e.g., summing, weighting, 
etc.) the importance values of each macroblock in the set. Priority is thus decided, for 
instance, based on cumulative importance of the macroblocks in each set. 

20 Importance analysis system 22 further comprises a partial macroblock analysis 

system 27 that analyzes macroblock importance when reference macroblocks do not 
exactly coincide with the current macroblock being analyzed (i.e., when a current 
macroblock acts a partial reference block). Specifically, in cases where only part of a 
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current macroblock is used as the reference macroblock, partial macroblock analysis 
system 27 computes the overlap (in terms of pixels) between the current macroblock and 
the reference macroblock. Thus, for example, if there were an overlap of 128 of 256 
pixels, the importance value would be scaled by 50%. An example of this is described 
5 below with respect to Figure 5. 

An exemplary algorithm for calculating an importance value of a macroblock in a 
P frame would be as follows: 

read a current macroblock; 

set importance_value (of the current macroblock) = 0; 
10 identify target macroblocks in B frames and the subsequent P frame that reference 

the current macroblock; 

for each identified target macroblock: 

examine the corresponding reference macroblock; 

if the reference macroblock exactly coincides with the current macroblock, 
1 5 then importance_value = importance_value + 1 ; 

else 

compute pixel overlap (maximum is 16x1 6=25 6); 
importance_value = importance_value + (pixel overlap/256); 

end. 

20 As an alternate embodiment, importance analysis system 22 may also include an 

indirect analysis system 29 that examines subsequent indirect predictions in determining 
importance. Indirect analysis system 29, while more computationally expensive, 
provides a more accurate valuation scheme. For example, in MPEG coding, 
macroblocks in I frames are used to "directly" predict P frame macroblocks, which in 

25 turn are used to "indirectly" predict subsequent P frame macroblocks and B frame 
macroblocks, and so forth. So in computing the importance of an I or a P frame 
macroblock, importance analysis system 22 may be used to not only examine direct 
predictions, but also examine subsequent indirect predictions. Thus, if an I macroblock 
acts as a reference for motion prediction for a macroblock in a P frame (direct 
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prediction), and subsequently the P frame macroblock acts as a reference for other 
macroblocks in subsequent P and B pictures (indirect prediction), the importance values 
of the indirect predictions can be added, or otherwise factored into the importance values 
of the direct prediction. Accordingly, the relative importance among and between 
5 macroblocks in both I and P frames can be computed and prioritization can be based on 
such results. 

As a further alternate embodiment, the importance value can be calculated (or 
further calculated) based on discrete cosine transform (DCT) residual values. A residual 
is the difference between a target macroblock and a reference macroblock. Thus, the 

10 smaller the residual, the closer the target macroblock matches the reference macroblock, 
and the greater the importance. Accordingly, residual analysis system 19 can examine 
the residual of each identified target macroblock and compute a function of each residual 
(e.g., the absolute or weighted sum of the coefficients). The importance value of the 
current macroblock can then be calculated based on, for example, a cumulative value of 

15 the residual computations from each target macroblock. It should be appreciated that this 
embodiment can be combined or used separately from the other embodiments described 
herein. 

Once the importance values have been obtained, stream prioritization system 24 
will assign priority to individual macroblocks or groups of macroblocks. In general, the 
20 greater the number of predictions for which the current macroblock is referenced by a 
target macroblock, the greater its priority, since the macroblock has a relatively higher 
importance for the visual quality of the decoded video stream. 
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In a preferred embodiment, since B frames are not used as references in predictive 
coding, the macroblocks in B frames are assigned the lowest priority. The macroblocks 
in the P frames are then assigned a relative higher priority, with the individual P frame 
macroblock data being prioritized in the manner described above. Finally, the 

5 macroblocks in the I frames are assigned the highest priority, again with the macroblock 
data being further prioritized based on the methods discussed above. 

Decoder 14 includes a stream decoding system 30 that comprises a correlation 
system 21 and an interpretation system 23. Correlation system 21 correlates each of the 
multi-priority streams 16 with a particular error protection scheme, and interpretation 

10 system 23 interprets the data and packages it accordingly. 

Referring now to Figure 2, an example of how an importance value is calculated 
for a P frame macroblock is shown. A stream of MPEG video data 32 is depicted 
comprised of a sequence of frames (P, B, B, P, B, B, P). In accordance with the methods 
discussed above, the macroblock data within P frame 33 is being analyzed for its 

15 importance. Specifically, a current macroblock 31 is examined (as shown by the arrows) 
to determine how often the current macroblock 31 acts as a reference macroblock for 
target macroblocks in previous and subsequent B frames 70, and target macroblocks in 
the subsequent P frame 34. As can be seen, current macroblock 31 acts as a reference 
macroblock for nine target macroblocks (shown as squares with a diagonal line). The 

20 target macroblocks could be any one of the 16 x 16 blocks (not shown) in the neighboring 
frames 70 and 34. Assuming exact coincidence between the current macroblock 31 and 
the corresponding reference macroblock, the macroblock would be assigned an 
importance value of nine. Accordingly, macroblock 31 would be assigned a relative 
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priority based on this value as compared to the other macroblocks in P frame 33. Note 
that in this case a subsequent P frame 34 follows P frame 33. In other cases (not shown), 
P frame 33 may be followed by an I frame, in which cases the subsequent I frame would 
not be analyzed for target macroblocks. 

5 Referring now to Figure 3, a similar example of how an importance value is 

calculated for an I frame macroblock is shown. In this case, a stream of frames 38 (P, B, 
B, I, B, B, P) is shown, and the macroblock data of I frame 36 is being analyzed to 
determine the relative priority of each macroblock. in I frame 36. Here again, target 
macroblocks are identified by examining the motion vectors in the subsequent P frame 

10 and neighboring B frames. In this case, there are eight target macroblocks that are 
predicted by the current macroblock. Assuming exact coincidence between current 
macroblock 35 and the corresponding reference macroblock, an importance value of eight 
would be assigned to the current macroblock 35. 

Referring now to Figure 4, an example is shown of how an importance value is 

15 calculated for an alternate embodiment utilizing indirect analysis system 29. 

Specifically, a stream of frames 40 (P, B, B, P, B, B, P) is shown, with the macroblock 
data in P frame 42 being analyzed to determine relative priority. It can be seen that a 
current macroblock 41 acts as a reference macroblock for a total of five target 
macroblocks in both B frame 44 and P frame 46. In addition, the target macroblock 43 in 

20 P frame 46 further acts as an "indirect" reference macroblock for a total of six indirect 

target macroblocks in B frame 48 and P frame 50. Assuming no other target macroblocks 
in P frame 46 act as reference macroblocks, the importance value for current macroblock 
41 of P frame 42 would be eleven. Although not shown, a more complex chain of 
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indirect calculations could be utilized. For example, the target macrob locks in P frame 
50 could be further examined to determine how often they act as reference macroblocks, 
etc. 

Referring now to Figure 5, an example is shown of how a partial reference block 

5 is taken into account by partial macroblock analysis system 27. Specifically, frame 52 
(e.g., I or P) includes a current macroblock 54 that is being analyzed for importance, a 
reference macroblock 56 that does not exactly coincide with current macroblock 54, and 
an overlap portion 60 that represents the portion where macroblock 54 and reference 
macroblock 56 coincide. In this example, only 25% of the pixels in the current 

10 macroblock 54 are used as a reference macroblock for target macroblocks in other 

frames. Thus, the importance value for this particular macroblock would be scaled (e.g., 
by 25%) to account for the overlap. 

It is understood that the systems, functions, methods, and modules described 
herein can be implemented in hardware, software, or a combination of hardware and 

1 5 software. They may be implemented by any type of computer system or other apparatus 
adapted for carrying out the methods described herein. A typical combination of 
hardware and software could be a general-purpose computer system with a computer 
program that, when loaded and executed, controls the computer system such that it 
carries out the methods described herein. Alternatively, a specific use computer, 

20 containing specialized hardware for carrying out one or more of the functional tasks of 
the invention could be utilized. The present invention can also be embedded in a 
computer program product, which comprises all the features enabling the implementation 
of the methods and functions described herein, and which - when loaded in a computer 


12 


* * 


system - is able to carry out these methods and functions. Computer program, software 
program, program, program product, or software, in the present context mean any 
expression, in any language, code or notation, of a set of instructions intended to cause a 
system having an information processing capability to perform a particular function 
5 either directly or after either or both of the following: (a) conversion to another language, 
code or notation; and/or (b) reproduction in a different material form. 

The foregoing description of the preferred embodiments of the invention have 
been presented for purposes of illustration and description. They are not intended to be 
exhaustive or to limit the invention to the precise form disclosed, and obviously many 
10 modifications and variations are possible in light of the above teachings. Such 

modifications and variations that are apparent to a person skilled in the art are intended to 
be included within the scope of this invention as defined by the accompanying claims. 
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